← All Tags
# inference optimization → quantization → large language models → gpu memory

1 link tagged with all of: inference optimization + quantization + large language models + gpu memory

Click any tag below to further narrow down your results

+ airllm (1)

Links

GitHub - lyogavin/airllm: AirLLM 70B inference with single 4GB GPU

AirLLM is a framework that streams model layers from disk one at a time during inference, letting you run massive language models like DeepSeek-V3 (671B parameters) on single consumer GPUs without quantization or distillation. It also supports fine-tuning huge models on small VRAM by keeping only adapters in memory while streaming frozen weights.
  • Runs a 671B parameter model on ~12GB VRAM and 125B models on ~6GB by keeping only one decoder layer on GPU at a time, with the rest streamed from disk
  • Works with virtually every major open-source model family (Llama, Qwen, DeepSeek, Mistral, Gemma, etc.) through a single `AutoModel.from_pretrained()` call with no special configuration
  • Supports optional 4-bit/8-bit weight quantization for up to 3x speedup, and fine-tuning of 125B models under 6GB VRAM by streaming base weights while keeping adapters resident
Last saved Oct 12, 2026 · 6 min read
+ airllm large language models inference optimization gpu memory quantization